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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1 ,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 
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•  Program  Executive  Officer,  Integrated  Warfare  Systems 
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Navy  submarine  acoustic  and  tactical  information  systems.  Bruhns  served  as  an  original  team  leader 
for  the  Acoustics  Rapid  Cots  Insertion  (ARCI)  program  office  with  an  emphasis  on  translating  fleet 
requirements  to  specifications,  implemented  using  commercial  off-the-shelf  (COTS)  hardware  and 
open  software  standards.  He  recently  led  a  successful  pilot  project  to  demonstrate  cross-program 
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operating  period  (MFOP)  demonstration  for  surface  ships. 

Abstract 

The  Deputy  Assistant  Secretary  of  the  Navy  for  Research,  Development,  Test,  and 
Evaluation  (DASN  RDT&E)  created  a  Business  Innovation  Initiative  (Bll)  to  identify  and 
overcome  challenges  presented  by  the  updated  Naval  Open  Systems  Architecture  (OSA) 
strategy.  The  Bll  seeks  to  find  innovative  ways  to  incentivize  Naval  program  management 
staff  and  their  industry  partners  to  implement  aggressive  change  measures  that  improve  cost, 
performance,  and  time  to  field  for  national  security  systems.  The  Bll  conducts  workshops  and 
crowd-sourcing  activities  that  focus  on  specific  business  impediments  to  OSA.  The  Massive 
Multiplayer  Operational  War  Game  Leveraging  the  Internet  (MMOWGLI)  game  was  used  as  a 
crowd-sourcing  tool  to  elicit  the  collective  intelligence  of  acquisition  practitioners,  students  of 
business,  and  industry  stakeholders  to  identify  and  overcome  challenges  presented  by  the 
updated  OSA  strategy.  The  MMOWGLI  platform  provides  an  efficient  mechanism  that 
crosses  functional  and  geographical  workspace  boundaries  for  exchanging  ideas  and  forming 
community  in  addressing  the  OSA  business  problem.  The  results  of  the  first  Bll  crowd¬ 
sourcing  event  using  the  MMOWGLI  platform  are  presented  in  this  report. 

Introduction 

The  Assistant  Secretary  for  Research  Development  and  Acquisition  (ASN[RDA]) 
authorized  a  new  Naval  Open  Systems  Architecture  (OSA)  strategy  in  November  2012  (see 
Appendix  A)  to  reduce  the  total  ownership  cost  of  systems,  encourage  innovation,  and  more 
rapidly  deliver  needed  capabilities  to  the  warfighter.  This  strategy  specifically  challenges  the 
Naval  acquisition  workforce  to  institute  measures  to  improve  competition,  eliminate 
redundant  developments,  and  coordinate  program  activities  that  promote  the  reuse  of 
tactical  products  across  sea  and  air  platforms.  The  acquisition  organization  is  tasked  to 
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implement  the  strategy,  and  success  will  require  substantial  changes  in  the  Navy’s  business 
practices,  organizational  structures,  and  resource  planning. 

In  concert  with  the  updated  strategy,  the  Deputy  Assistant  Secretary  of  the  Navy  for 
Research,  Development,  Test,  and  Evaluation  (DASN  RDT&E)  created  a  Bll  to  search  for 
ways  to  better  incentivize  OSA  business  practices  across  our  current  programs  of  record. 
Mr.  Sean  Stackley,  ASN(RDA),  said  in  a  recent  article, 


The  value  of  an  innovation  initiative  is  to  explore  what  business  relationship 
changes  are  needed  to  open  up  competition;  incentivize  better  contractor 
performance;  increase  access  to  innovative  products  and  services  from  a 
wider  array  of  sources;  decrease  time  to  field  new  capabilities;  and  achieve 
lower  acquisition  and  life-cycle  costs  while  sustaining  fair  industry  profitability. 
(Hudson,  2012) 


The  Updated  Naval  OSA  Strategy  and  the  Bll 
Relationship 
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Figure  1.  Updated  Naval  Open  Systems  Architecture  Strategy  and 
Business  Innovation  Initiative  Relationship 


As  a  companion  of  the  updated  Naval  OSA  strategy,  the  Bll  examines  business 
operations,  processes,  and  incentives  associated  with  delivering  national  security  systems. 
DASN  RDT&E  recognized  that  using  the  crowd-sourcing  war  game  facility  developed  by  the 
Naval  Postgraduate  School  (NPS)  presented  an  opportunity  to  examine  both  the  OSA 
strategy  and  the  Bll  in  tandem.  Both  share  activities  that  could  be  explored  and  melded 
together  into  a  comprehensive  set  of  recommendations  on  how  to  advance  technical  and 
business  changes  needed  in  Naval  acquisition. 
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The  Bll  addresses  a  two-year  plan  that  will  codify  methods  to  rapidly  bring  innovative 
capabilities  to  the  warfighter  and  find  a  process  that  seeks  continuous  cost  improvements 
for  initial  acquisition,  production,  and  sustainment,  while  fostering  innovation  from  a  broad 
range  of  sources. 

In  the  first  year,  the  Naval  OSA  Enterprise  Team  (ET)  is  promoting  business  models 
that  lower  overall  cost  and  ensure  a  reasonable  profit  potential,  reduce  time  to  market,  and 
remove  competitive  restrictions.  This  innovation  initiative  is  looking  for  reforms  that  create 
effective  business  relationships  between  government  and  industry  (government-to-business) 
and  between  government  organizations  (government-to-government). 


Naval  OSA  Priorities: 

Rapid  Capabilities  to  the  Warfighter. 
Continuous  Cost  Improvement. 

Increased  Innovation  from  a  Broad  Range  of 
Sources 


Promote  Business  Models  That: 
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--Reduce  Tone  to  Market 
—Remove  Competitive  Restrictions 


Government  to  Business  Reform 
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Government  OSAET 


Figure  2.  Naval  Open  Systems  Architecture  Business  Innovation  Intiative 

Government  to  Business:  In  the  first  year,  the  government-to-business  team  is 
acting  to  create  an  open  business  model  framework  that  recognizes  that  the  defense  market 
is  changing  and  seeks  to  find  opportunities  to  improve  and  normalize  business  relationships. 
Team  1  consists  of  academia  (top-tier  business  schools  and  NPS),  industry  representatives 
(strategically  formed  to  have  a  balanced  representation  with  participation  from  large  and 
small  businesses),  and  key  Department  of  the  Navy  (DoN)  acquisition  staff.  The  team  is 
scheduled  to  complete  its  objectives  within  12  months  and  publish  its  findings  and 
recommendations  to  the  ASN(RDA). 

Government  to  Government:  The  second  team  will  complement  the  work  of  the 
first  team  and  will  focus  on  changes  that  affect  the  government  acquisition  community. 

Team  2  will  consist  of  key  representatives  from  the  resource  community,  the  fleet,  and  the 
acquisition  force  (PEOs,  SYSCOMs,  and  laboratories).  The  team  will  deliver  actionable 
supporting  policies,  new  procedures,  workforce  education  /outreach  tools,  and  candidate 
legislative  changes. 

Bll  War  Games 

The  term  “war  game”  is  used  to  describe  the  crowd-sourcing  concept  exploration 
events  of  the  Bll.  The  war  game  is  a  social  networking  construct  to  explore  and  solve 
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“wicked  problems”  (Conklin,  2005)  in  a  large  and  diverse  forum.  Wicked  problems  are  those 
with  complex  interdependencies  where  the  effort  to  solve  the  stated  problem  may  reveal  or 
create  other  problems.  In  Naval  OSA,  for  example,  a  recognized  problem  is  “vendor  lock.” 
Vendor  lock  is  the  business  principle  associated  with  limited  acquisition  choice  and  sole- 
source  contracting  and  may  further  be  an  indication  of  an  unrecognized  strategic  problem. 

The  DASN  RDT&E  selected  the  Massive  Multiplayer  Operational  War  Game 
Leveraging  the  Internet  (MMOWGLI)  as  the  mechanism  to  bring  innovative  concepts  from  a 
diverse  group  together  rapidly  and  effectively.  MMOWGLI  is  an  open-source  software 
platform  sponsored  by  the  Office  of  Naval  Research  (ONR),  developed  by  NPS,  and 
originally  designed  by  Institute  For  The  Future  (IFTF).  MMOWGLI  provides  a  text-based 
social  networking  platform  that  allows  many  users  to  interact  directly  with  one  another  using 
web  browsers  in  real  time. 

The  first  Bll  MMOWGLI  game  was  intended  to  serve  as  a  trial  run  for  understanding 
whether  a  crowd-sourcing  method  for  exploring  innovations  for  implementation  of  the  Naval 
OSA  strategy  might  be  valuable  and  robust.  Round  One  of  the  Bll  MMOWGLI  game  is 
described  in  detail  in  Section  2. 
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Interactivities 


Bll  MMOWGLI  Game 
What  Is  MMOWGLI? 

MMOWGLI  is  an  online  game  platform  designed  to  elicit  collective  intelligence  from 
an  engaged  pool  of  players  to  solve  real-world  problems.  MMOWGLI  was  developed  at  the 
NPS’s  MOVES  Institute  under  the  sponsorship  of  the  ONR.  It  was  launched  in  201 1 . 
MMOWGLI  games  have  been  conducted  to  look  for  innovative  ideas  in  a  variety  of  complex 
problems,  including  countering  international  maritime  piracy,  improving  energy  efficiency  on 
Navy  ships,  and  several  others.  Games  have  also  been  conducted  at  a  variety  of  scales  (50 
to  550  players)  and  with  different  audiences  (public,  industry,  Navy,  and  others). 
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Bll  MMOWGLI  Round  One 

War-game  design  is  an  important  part  of  the  process.  The  MMOWGLI  platform  is 
designed  to  spur  innovation  and  is  highly  configurable,  but  players  will  not  visit  or  engage 
unless  a  clear  purpose  is  evident.  Several  months  of  effort  went  into  defining  and  refining 
the  game  themes  and  prospective  audience  that  might  provide  the  greatest  possible  impact 
supporting  the  overall  project  goals.  A  preliminary  workshop  and  a  trial  mini-game  provided 
excellent  feedback  regarding  what  themes  were  most  interesting.  Extensive  details  and  the 
game  itself  are  available  online  (see  Bll  MMOWGLI  Game  Portal,  n.d.;  Bll  MMOWGLI  Game 
Blog,  n.d.;  Bll  MMOWGLI  Game  Play,  n.d.;  Bll  MMOWGLI  Game  Reports,  n.d.). 

The  DASN  RDT&E  conducted  the  first  of  two  planned  Bll  MMOWGLI  games  for 
fiscal  year  (FY)  2013  during  the  period  of  January  14  through  28.  The  purpose  of  the  first 
round  was  to  test  and  validate  the  use  of  the  MMOWGLI  crowd-sourcing  tool  for  finding 
innovation  in  Naval  acquisition  and  to  identify  how  to  use  it  to  find  challenges  of 
implementing  the  updated  Naval  OSA  strategy.  Over  100  professionals  from  government, 
academia,  and  industry  participated,  generating  over  890  idea  cards  and  1 1  action  plans. 
The  first  week  was  dedicated  to  card  play,  followed  by  a  second  week  used  for  detailed 
action-plan  development. 

The  Call  to  Action  Statement 

Establishing  group  motivation  and  common  goals  is  a  fundamental  prerequisite  for 
an  effective  war  game.  The  Bll  MMOWGLI  game  was  introduced  to  players  with  this 
invitation: 


The  BII  MMOWGLI  game  is  for  professionals  exploring  how  best  to  achieve  the 
business  goals  of  the  Navy’s  new  “Open  Systems  Architecture  (OSA)  Strategy.”  Your 
feedback  on  this  plan  is  welcome. 

Round  One  explores  the  challenges  and  opportunities  in  developing  a  “Payloads  over 
Platforms”  business  model  driven  by  the  OSA  strategy.  We  are  here  to  discover  ways  to 
incentivize  acquisition  programs  and  our  industry  partners  to  help  craft  a  new  business 
relationship  that  eliminates  redundant  development,  ensures  sustainability  of  war-fighting 
capabilities,  and  positively  rewards  good  industry  performance. 

Your  contributions  are  essential.  Please  join  in.  The  BII  portal  is  a  great  information 
resource  for  game  play.  Check  the  BII  blog  for  game  news.  Thanks  for  your  ideas. 

Play  the  game,  change  the  game. 


Card  Play 

All  MMOWGLI  games  begin  with  a  set  of  thought-provoking  seed  cards  to  initiate 
discussion.  In  the  Bll  game  Round  One,  the  seed  cards  were  labeled  as  “challenges.”  Each 
challenge  card  was  configured  to  provide  four  responses:  expand,  counter,  adapt,  or 
explore.  Players  would  select  a  response  category  and  type  their  thoughts  in  140  characters 
maximum,  which  forced  them  to  be  clear  and  concise  with  their  message.  Other  players 
could  immediately  see  that  card  as  part  of  the  discussion  chain  and  be  able  to  select  it  and 
respond  in  kind:  expand,  counter,  adapt,  or  explore.  These  linked  cards  created  a  chain  (or 
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discussion)  where  any  number  of  players  could  contribute.  Rules  on  how  to  play  a  particular 
response  card  were  left  to  the  player’s  intuition.  See  Figure  4. 
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Challenge  Cards.  In  our  Bll  MMOWGLI  game,  seven  general  questions  about  the 
strategy  were  cast  as  challenge  cards,  intended  to  help  players  begin  conversations  of 
interest.  They  were  as  follows: 


Challenge  Cards: 

1 .  What  are  the  primary  challenges  of  the  CNO’s  “payloads  over  platforms” 
strategy? 

2.  How  to  incentivize  programs  and  industry  to  deliver  reusable  component 
solutions  as  an  acquisition  model? 

3.  How  can  we  align  across  programs  to  eliminate  redundant  development? 

4.  How  can  Naval  acquisition  align  programs  to  a  limited  number  of  technical 
reference  frameworks  (TRFs)? 

5.  Technology  conditions  change  over  long  service  lives  of  ships  and  aircraft. 

6.  Has  open  architecture  (OA)  succeeded  from  a  micro  perspective  and  failed  in 
the  macro  perspective? 

7.  What  do  you  see  as  the  greatest  challenge  moving  into  Phase  1  of  the  OSA 
strategy? 

An  additional  set  of  seven  seed  cards  were  also  formed  under  the  heading  Future 
Goals.  They  were  “what  do  you  think?”  topics  intended  to  stimulate  discussion  on  system 
acquisition.  They  were  as  follows: 
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Future  Goals  Cards: 


1 .  How  might  the  acquisition  process  change  to  enable  more  competition  by 
industry:  large  companies,  small  companies,  and  system  integrators? 

2.  How  can  life  cycle  management  (LCM)  improve  to  avoid  “lock  in”  and  enable 
competition  through  the  long  term  program? 

3.  What  ships  and  aircraft  are  the  best  exemplar  platforms  to  consider? 

4.  What  Navy  programs  are  the  best  exemplar  payloads  to  consider? 

5.  Unmanned  systems  have  much  shorter  lifecycles,  enabling  rapid  change. 
How  might  that  improve  the  acquisition  process? 

6.  How  do  we  define  payloads?  One  person’s  payload  is  another  person’s 
truck... 

7.  What  topic  do  you  want  to  see  addressed  in  round  2  of  the  Game  next 
summer?  Our  theme:  Future  Paths  Forward  For  Improved  Business 
Practices. 


Game  Play  Results 

Figures  5  and  6  show  the  number  of  player  responses  for  each  of  the  seed  cards. 
Expand  and  explore  cards  were  most  often  chosen  by  the  players  to  amplify  on  a  previous 
card.  Counter  and  adapt  card  responses  were  given  much  less  frequently. 
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Future  Goals  Card  Chains 
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Figure  6.  Future  Goals  Card  Chain 


Four  seed  cards  generated  the  most  response.  It  is  noted  that  these  cards  focused 
more  on  Phases  2  and  3  of  the  OSA  strategy. 

Challenge  Seed  Card  #2:  How  to  incentivize  programs  and  industry  to  deliver 
reusable  component  solutions  as  an  acquisition  model? 

Finding  ways  to  incentivize  programs  and  industry  to  deliver  reusable  component 
solutions  as  an  acquisition  model  generated  the  second  highest  interest  in  the  game  with 
151  total  exploration  cards  played.  Today,  acquisition  programs  most  often  employ  effort- 
based  development  contracts  for  a  single  end  user  without  considering  how  it  might  be 
reused  in  other  programs.  A  new  concept  of  a  “first  user-subsequent  user”  model  was 
offered  as  a  means  to  include  the  subsequent  user  in  the  acquisition  model  where  vendor 
savings  naturally  occur,  indicating  how  the  Navy  and  industry  might  share  benefits  of  reuse. 
Conversations  focused  on  data  rights,  intellectual  property,  licensing,  royalties,  and  RFP 
process  areas.  Industry  has  the  incentive  of  an  expanded  market,  and  government  has  the 
incentive  of  reduced  schedule  and  cost  profiles. 

Challenge  Seed  Card  #3:  How  can  we  align  across  programs  to  eliminate 
redundant  development? 

This  topic  drew  the  highest  response  in  the  game  with  157  exploration  cards  played. 
Program  objectives  memoranda  (POM)  roadmaps  are  a  resource  management  tool  that 
align  program  funding  to  the  corresponding  Future  Years  Defense  Program  (FYDP). 
Financial  plans  for  product  line  acquisitions  were  offered  as  a  way  to  show  sponsors  that 
collaboration  equals  value.  These  would  include  first  user-future  user  program 
budget/schedules  as  a  way  to  help  PEOs  and  resource  sponsors  connect  the  dots. 

Challenge  Seed  Card  #7:  What  do  you  see  as  the  greatest  challenge  in  executing 
Phase  1  of  the  OSA  strategy? 
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One  player  cited  Dr.  Gansler’s  “The  Changing  Face  of  Competition,”  noting  that  one 
third  of  the  acquisition  workforce  has  less  than  five  years  of  experience  and  does  not 
understand  industry  operations  and  incentives.  As  a  result,  government  relies  on  a 
contracting  style  based  more  on  precedence  (i.e.,  how  we  do  things  today  rather  than  how 
we  might  do  things  better).  Players  focused  on  the  need  for  specific  workforce  training  for 
OSA  as  the  greatest  challenge  to  the  strategy. 

Challenge  Seed  Card  #4:  How  can  Naval  acquisition  align  programs  to  a  limited 
number  of  technical  reference  frameworks  (TRFs)? 

Citing  declining  budgets  as  a  motivator,  it  was  noted  that  some  PEOs  are  starting  to 
coordinate  technical  frameworks  into  their  plans  and  programs.  Even  though  PEOs 
collaborate,  there  is  no  technical  chain  of  command,  so  aligning  frameworks  becomes  a 
coalition  of  the  willing.  Players  suggested  having  the  government  own  their  TRFs,  so 
product  developers  have  a  stable  base  to  enable  broader  competition.  A  counterargument 
stated  that  “a  limited  number  should  not  be  the  goal  since  different  programs  have  different 
needs.”  This  was  quickly  challenged  by  another  player:  “A  different  need  is  code  for  not 
invented  here.”  The  message  from  this  discussion  thread  is  that  coordinated  TRF  design 
and  governance  will  form  a  key  role  in  the  OSA  strategy. 

Action  Plan  Products 

Idea  card  chains  quickly  illustrate  the  pros,  cons,  and  alternatives  associated  with  an 
issue.  More  is  needed  when  moving  beyond  improved  understanding  towards  charting 
possible  paths  forward  and  analyzing  alternative  courses  of  action. 

In  MMOWGLI,  an  action  plan  is  an  expansion  of  an  idea  formed  in  game  play.  It  is 
authored  by  a  small  group  of  players,  usually  three  or  four,  who  collaborate  to  describe  how 
an  idea  might  work,  what  the  benefits  are,  and  the  scope  of  effort  necessary.  There  are  five 
parts  to  an  action  plan  (essentially,  who,  what,  where,  when,  why  and  how).  Below,  the  “how 
will  it  work”  section  is  included  as  a  summary  of  the  plan. 

The  following  five  action  plans  were  significant  in  that  they  achieved  a  greater  than 
66%  (two  of  three  possible  thumbs-up)  voter  rating.  Players  that  vote  one  thumb-up  think 
that  the  plan  is  not  ready  to  proceed.  Players  that  give  two  thumbs-up  consider  an  action 
plan  worthwhile,  while  players  awarding  three  thumbs-up  mean  that  they  think  that  a  plan  is 
especially  promising  and  worthy  of  further  exploration.  Here  are  the  top  five  plans,  based  on 
averaged  player  ratings,  from  highest  to  lowest: 

Action  Plan  #10:  How  can  the  Navy  use  data  and  software  rights  to  achieve  the 
OSA  strategy,  ensuring  long-term  stability  and  growth? 

How  will  it  work?:  Conduct  a  series  of  workshops  and  discussion  groups  to  focus  on 
different  perspectives,  including  how  we  best  leverage  the  commercial  market  (while 
addressing  issues  and  challenges  unique  to  DoD  systems)  and  how  we  establish  a 
marketplace  for  reusable  applications  to  reduce  cost  and  increase  innovation  and 
interoperability.  Result  is  a  spectrum  of  strategies  from  which  programs  can  select 
depending  on  the  reuse  potential  of  specific  products  in  identified  markets.  Then  train  the 
workforce. 

Action  Plan  #3:  Define  community  of  interest  (COI)  activities  and  external 
relationships  in  the  context  of  the  OSA  strategy. 

How  will  it  work?:  The  main  catalyst  for  COIs  today  is  interoperability  for  mission 
performance,  rather  than  reuse.  The  mission  area  interoperability  and  integration  (l&l)  effort 
should  help  define  systems  of  systems  (SoS)  capability  gaps.  Once  people  start  thinking  in 
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SoS  terms,  they  can  start  looking  at  reuse  across  platforms.  That  often  leads  to  a 
homegrown  approach  and  a  PARM  relationship  between  programs.  PEOs  submit  capability- 
based  coordinated  POM  inputs  for  whole  capabilities.  The  OPNAV  coordinates  funding 
decisions  so  that  holistic  upgrades  are  funded. 

Action  Plan  #1 1 :  Streamline  OSA  contracting  to  make  the  process  more  agile, 
rewarding  innovation. 

How  will  it  work?:  Unlike  the  SBIR  program,  all  efforts  would  be  fully  funded,  but  they 
would  be  similar  to  SBIRs  in  that  they  are  incrementally  funded.  Initial  awards  based  on  start 
year  (e.g.,  2014  base  year,  with  option  years  1  ...  5).  Base-year  candidates  would  be 
minimally  funded  with  data  rights  and  royalties  for  reuse,  while  more  mature  option  year 
awards  would  benefit  from  increased  funding.  This  would  push  competition  for  best  solutions 
to  win  the  options  as  well  as  the  additional  royalties. 

Action  Plan  #8:  Outline  ROI  steps  to  justify  reusing  components  across  participating 
COI  programs. 

How  will  it  work?:  An  ROI  metric  would  quantify  the  different  ways  we  save  via  reuse, 
including  requirements  specification,  integration,  test,  training,  maintenance,  sparing  (for 
H/W),  IAVM,  and  technology  refresh.  COPLIMO  has  a  COCOMO-based  cost  model  that  can 
look  at  SW  product  lines  and  show  where  costs  are  reduced  with  subsequent  reuse  by 
leveraging  existing  requirements.  Also,  we  need  a  way  to  quantify  operational  benefits  from 
consistency  of  performance,  user  interfaces,  and  more  widespread  fielding. 

Action  Plan  #6:  How  do  programs  that  group  together  get  rewarded  for  increasing 
enterprise  value? 

How  will  it  work?:  In  this  budget  climate,  PMs  will  seek  protection,  even  at  the  cost  of 
increased  program  risk,  by  coupling  budget  and  schedules  together.  Top  leadership 
direction  will  be  needed  to  ensure  that  the  reward  mechanisms  are  impactful  and 
consistently  applied.  Enterprise  value  must  be  held  to  a  very  high  and  consistent  standard. 
Accolades  must  be  peer-reviewed.  This  would  have  two  benefits:  the  attention  to  award 
criteria  will  rise,  and  consistency  of  awards  will  increase. 

Lexical  Link  Analysis 

Results  were  analyzed  independently  at  NPS  using  the  lexical  link  analysis  (LLA) 
software-based  text  comparison  tool.  LLA  compared  Bll  game  data  to  the  OSA  strategy 
document  and  separately  to  the  Department  of  Defense  (DoD)  Open  System  Architecture 
Contract  Guidebook  for  Program  Managers  (Defense  Acquisition  University  [DAU],  n.d.). 

LLA  metrics  indicate  that  the  OSA  strategy  is  not  considered  risky  and  not  particularly 
controversial,  nor  is  it  considered  impossible  to  implement. 
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What  does  LLA  indicate  about  bii  MMOWGLI  Round  1? 

•  LLA  analysis  of  MMOWGLI  data  indicates 

—  Ideas  and  draft  Action  Plans  expressed  in  bii  game,  by 
anonymous  players,  showed  strong  consistency  with  the 
concepts  in  Program  Manager's  Contract  Guidebook 

—  Metrics  indicate  draft  OSA  strategy  triggered  new  and 
innovative  ideas 

—  Metrics  did  not  indicate  that  OSA  strategy  was  risky, 
controversial,  or  impossible  to  implement  etc. 

•  Next-round  game  efforts: 

—  Focus  on  LLA-discovered  themes  of  particular  interest 

—  Encourage  even  more  player  input 


Figure  7.  LLA  Conclusions,  BII  MMOWGLI  Analysis  Report,  February  6,  2013 

LLA  is  a  form  of  text-based  data  mining  in  which  word  meanings  represented  in 
coupled  lexical  terms  (e.g.,  word  pairs)  can  be  represented  as  if  they  are  in  a  community  of 
a  word  network.  LLA  discovers  and  displays  these  networks  of  word  pairs  from  large-scale, 
unstructured  data.  LLA  can  also  be  used  as  a  search  and  knowledge  management  tool  for 
scoring  and  ranking  interesting  information  and  for  visualizing  and  reporting  correlations 
among  categories  and  layers  of  information  including  lexical,  semantic,  and  social  links. 

This  effort  then  presents  the  decision-maker  with  results  that  were  previously 
unavailable  regarding  emerging  patterns  and  themes,  as  well  as  unprecedented  levels  of 
analysis,  thus  reducing  the  workload  and  overcoming  the  blind  spots  of  human  analysts  by 
leveraging  automation.  For  example,  for  the  recent  MMOWGLI  games  used  to  develop  and 
identify  new  ideas  about  a  specific  subject,  LLA  was  engaged  to  identify  potentially 
interesting  information  from  idea  cards,  link  them,  and  then  reveal  them  to  domain  experts. 
The  methodology  of  LLA  is  discussed  in  more  detail  in  the  article  in  these  same  conference 
proceedings  (Zhao,  Brutzman,  &  MacKinnon,  2013). 

For  the  BII  MMOWGLI  project,  we  performed  two  separate  post-game  data  analyses. 

1 .  Idea  cards  (892)  and  action  plans  (11)  were  compared  to  the  proposed  OSA 
strategy  (four  pages)  considered  by  players. 

2.  Idea  cards  (892)  and  action  plans  (1 1 )  were  compared  to  the  Department  of 
Defense  (DoD)  Open  System  Architecture  Contract  Guidebook  for  Program 
Managers  (158  pages)  already  familiar  to  most  players. 

What  did  LLA  indicate  about  BII  MMOWGLI  Round  One?  LLA  data  analysis  indicates 
the  following: 
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•  Ideas  and  draft  action  plans  expressed  in  the  Bll  game  by  anonymous 
players  showed  strong  consistency  with  the  concepts  in  Department  of 
Defense  (DoD)  Open  System  Architecture  Contract  Guidebook  for  Program 
Managers. 

•  Metrics  indicate  that  draft  OSA  strategy  triggered  new  and  innovative  ideas. 

•  Metrics  did  not  indicate  that  OSA  strategy  was  risky,  controversial,  nor 
impossible  to  implement. 

LLA  also  discovered  eight  main  or  popular  themes  listed,  reflecting  common  interest 
of  the  players,  using  the  following  keywords: 

•  multiple  support  and  components 

•  common  data,  data  model 

•  component  reuse,  OSA 

•  open  system  and  business 

•  systems  architecture,  current  systems 

•  specific  price  and  fee 

•  existing  reusable  programs 

•  engineering,  government,  and  community 

We  also  found  innovative  ideas  (i.e.,  gaps  between  the  game  data  and  the  OSA 
strategy  document)  in  the  following  areas  (themes): 

•  small  and  shared  based 

•  developed  and  built  faster 

•  critical  definition 

•  specific  price  and  fee 

•  sponsors  change  and  risk 

•  changing  requirements 

•  interoperability  and  interfaces 

Figure  8  shows  one  example  theme  detailed  from  the  comparison  of  game  data  with 
the  OSA  strategy  document.  Red  nodes  show  the  top  three  word  hubs  with  most  links  (most 
“central”).  Yellow  word  pairs  are  unique  to  the  action  plans,  green  word  pairs  are  unique  to 
the  ideas  cards,  and  blue  word  pairs  are  unique  to  the  OSA  strategy  document.  Red  word 
pairs  are  found  in  more  than  two  sources. 
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Figure  8.  One  Theme  Detailed  From  the  Comparison  of  the  Game  Data  With  the  OSA 

Strategy  Document 

Note.  Red  nodes  show  the  top  three  word  hubs  with  most  links  (most  “central”).  Yellow  word  pairs  are 
unique  to  the  action  plans,  green  word  pairs  are  unique  to  the  ideas  cards,  and  blue  word  pairs  are 
unique  to  the  OSA  strategy  document.  Red  word  pairs  are  found  in  more  than  two  sources. 

Subject-matter  expertise  is  necessary  to  determine  whether  statistically  generated 
concepts  have  real  relevance  and  interest.  Based  on  the  concepts,  themes,  and  gaps 
discovered  in  Bll  Round  One,  analysts  compiled  and  distilled  a  list  of  20  candidate  topics  for 
possible  seed  card  exploration  in  Bll  Round  Two.  One  example  is  shown  as  follows,  using 
the  theme  illustrated  in  the  preceding  figure  and  centered  around  the  keywords  multiple- 
support  and  components. 

•  How  will  the  concept  of  TRF  addressed  in  the  strategy  document  be 
enhanced  from  the  ideas  surrounding  “multiple  support  and  components,”  a 
COTS-based  model? 

•  Relevant  concepts  from  Round  One:  Network-based  COTS  (Idea  Card  Chain 
850),  COTS  hardware  (Idea  Card  Chains  138,  89),  COTS  insertion  (Idea 
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Card  Chain  770),  COTS  model  (Action  Plan  4),  and  COTS  elements  (Action 
Plan  10) 

The  reason  emerging  concepts  such  as  this  can  be  generated  is  that  salient  ideas 
emerge  from  commonality  and  differences  between  data  corpus  comparisons,  allowing 
expert  analysts  to  discern  which  topics  might  hold  the  greatest  interest. 

A  number  of  reports  and  analytic  products  are  generated  by  the  game.  To  reap  the 
maximum  benefits  from  the  contribution  of  so  many  focused  ideas  and  plans,  post-game 
analysis  is  an  important  activity  that  helps  find  the  most  valuable  conclusions  and  also 
results  in  products  of  greatest  value  to  past  (and  future)  participants. 

Lessons  Learned 

Round  One  Results  Exceeded  Expectations.  The  DASN  RDT&E  expected  that  Bll 
MMOWGLI  Game  Round  One  would  generate  a  list  of  challenges  for  the  updated  OSA 
strategy.  Players  instead  concentrated  their  activities  on  identifying  ways  to  make  the 
strategy  successful.  It  was  clear  that  they  viewed  their  role  as  problem  solvers  rather  than 
problem  identifiers.  This  is  a  valuable  lesson  for  planning  Round  Two  of  the  Bll  MMOWGLI 
game.  Understanding  a  player’s  natural  motivation  to  create  ideas  on  challenges  gives  the 
planning  team  a  better  understanding  of  how  to  select  and  “seed”  the  topics  for 
consideration  in  the  next  round  of  play. 

Crowd  Sourcing  Builds  Community.  An  unexpected  benefit  from  using  this  tool 
was  the  aspect  of  community  building.  Those  who  played  Bll  MMOWGLI  Game  Round  One 
formed  a  diverse  team  with  different  points  of  view.  Game  players  were  anonymous  and 
used  a  made-up  player  name  to  engage  in  discussions.  The  anonymity  provided  a  shield 
from  intimidation  or  prejudice,  in  that  all  seemed  open  to  the  ideas  and  comments  of  others 
and  willing  to  engage  in  debate  on  the  pure  basis  of  the  issues,  rather  than  taking  an 
industry  or  government  positional  argument.  Players  expanded  their  ideas  using  a  game 
feature  called  action  plans,  inviting  other  players  to  help  them  author  potential  solutions  ... 
and  it  worked.  The  lesson  learned  here  is  to  open  the  problem  set  to  the  widest  audience 
possible.  Opinions  and  interactions  ultimately  come  together  to  build  the  views  of  a  larger 
community. 

Enlisting  a  Broad  Audience  Requires  Promotion  and  Advertising.  Bll 

MMOWGLI  Game  Round  One  was  as  much  about  testing  the  tool  as  it  was  about  getting 
direct,  actionable  results.  As  such,  marketing  was  minimal.  E-mail  invitations  shortly  ahead 
of  the  event  and  a  few  public  announcements  formed  the  ad  campaign.  Current  members  of 
the  OSA  ET,  Bll  academic  partners,  and  selected  industry  partners  who  volunteered  to 
participate  at  the  Defense  Daily  Open  Architecture  Summit  of  October  2012  formed  the  main 
body  of  invitees.  In  response  to  approximately  900  e-mail  invitations,  just  over  100  people 
played. 

Conclusions 

The  success  of  the  new  Naval  OSA  strategy  relies  on  the  Navy  acquisition 
community’s  ability  to  cross  program  boundaries  and  work  together  toward  common  goals. 
We  can  take  full  advantage  of  the  power  of  community  by  using  focused  information 
technology.  MMOWGLI  has  proved  to  be  an  effective  mechanism  to  improve  communication 
and  coordination  among  the  various  program  offices,  program  executive  offices,  and 
systems  commands.  The  DASN  RDT&E  considers  the  Bll  MMOWGLI  Game  Round  One  a 
success  for  the  following  reasons: 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-679- 


1 .  The  use  of  MMOWGLI  to  explore  Naval  OSA  challenges  exceeded 
expectations.  The  players  of  the  Bll  war  game  identified  several  innovations 
on  how  to  implement  the  Naval  OSA. 

2.  Game  play  supported  the  intent  of  the  Bll  to  explore  the  business 
relationships  changes  to  (a)  open  up  competition,  (b)  incentivize  better 
contractor  performance,  (c)  increase  access  to  innovative  products  and 
services  from  a  wider  array  of  sources,  (d)  decrease  time  to  field  new 
capabilities,  and  (e)  achieve  lower  acquisition  and  life-cycle  costs  while 
sustaining  fair  industry  profitability. 

3.  Those  who  played  Bll  MMOWGLI  Game  Round  One  formed  an  enthusiastic 
group  with  different  points  of  view,  highlighting  the  power  of  crowd  sourcing  to 
build  a  diverse  community  around  topics  of  mutual  interest. 

Multiple  recommendations  for  future  work  are  now  being  pursued  as  the  team 
prepares  for  Bll  MMOWGLI  Round  Two.  Our  motto  remains:  Play  the  game,  change  the 
game! 
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Appendix  A 


O-S-A  Open  Systems  Architecture  Strategy 


Introduction 

The  current  Naval  Enterprise  acquisition  model  is  centered  on  highly  integrated  platforms  with 
systems  that  are  largely  vendor  locked,  and  expensive  to  acquire  and  upgrade.  This  model  is  especially 
problematic  in  the  current  economic  environment. 

The  Naval  Open  Systems  Architecture  (OS A)  strategy  will  decompose  monolithic  business  and 
technical  designs  into  manageable  product  lines  composed  of  competition- driven  modular  Enterprise 
components.  This  will  yield  innovation,  reduced  cycle  time,  and  lower  total  ownership  costs. 

The  New  Naval  Enterprise  OSA  Strategy: 

The  Naval  OSA  Strategy  is  an  iterative  set  of  business  and  technical  changes  that  points  to 
an  end  state  where  affordable,  open  platforms  easily  accommodate  open  modules.  As  the  Navy 
moves  toward  this  future,  the  Enterprise  must  first  align  itself  to  become  open,  modular,  common, 
competitive,  and  ultimately,  affordable.  It  will  begin  by  implementing  change  in  a  coordinated 
fashion  across  all  programs. 

The  Naval  OSA  Enterprise  Team  will  lead  the  execution  of  this  strategy  with  the  participation  of 
stakeholders  (e.g..  Resource  Sponsors,  PEOs,  TWAs,  etc.)  as  follows: 

•  Implement  the  coordinated  set  of  business  changes  that  improve  competition,  incentivize 
better  performance,  and  deliver  capability  more  rapidly; 

•  Construct  a  limited  number  of  technical  reference  frameworks  to  immediately  support 
improved  competition  and  ultimately  enable  enterprise  re-use; 

•  Develop  an  Execution  Guidebook  for  this  strategy;  and 

•  Lead  and  guide  training  the  workforce  on  OSA  implementation. 

Once  these  changes  have  been  adopted  at  the  program  level  a  second  iteration  (Eigure  1)  will  prepare 
the  Enterprise  to  eliminate  redundancy  and  deliver  open  systems  with  reusable  modules. 


Figure  1.  Iterative  Naval  OSA  Strategy 


Open  Systems  Architecture  (OSA)  Strategy 
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An  Open  Systems  Architecture  (OSA)  approach  integrates  business  and  technical  practices  that 
yield  systems  with  severable  and  compete- able  modules.  A  system  constructed  in  this  way  allows 
vendor- independent  acquisition  of  warfighting  capabilities,  including  the  intentional  creation 
of  inter- operable  Enterprise- wide  reusable  components.  Successful  OSA  acquisitions  result  in 
reduced  total  ownership  cost  and  can  be  quickly  customized,  modified,  and  extended  throughout 
the  product  life  cycle  in  response  to  changing  user  requirements. 


Naval  OSA  Strategy  Actions 

Unless  otherwise  stated,  the  principal  lead  for  the  following  actions  is  the  Open  Systems 
■  Architecture  Enterprise  Team  (OSAET),  led  by  Deputy  Assistant  Secretary  of  the  Navy  (Research, 
Development,  Test  and  Engineering)  (DASN  RDT&E),  with  participation  from  Systems  Commands 
(NAVSEA,  NAVAIR,  SPAWAR,  and  MARCORSYSCOM),  program  Executive  Offices 
(PEO),  the  Office  of  Naval  Research,  and  designated  Communities  of  Interest  (COIs). 


Phase  1:  Align  current  programs  to  execute  the  OSA  strategy  and  report  progress. 


Phase  2:  Consolidate  technical  frameworks  across  programs;  eliminate  redundant  stovepipes. 
Phase  3:  Implement  Enterprise  architecture  of  modular  development  and  maximum  reuse. 

Phase  1  Tasks 

•  Establish  a  limited  number  of  OSA  Technical  Reference  Frameworks  (TRFs); 

•  Change  acquisition  processes  to  adopt  these  coordinated  and  common  OSA  TRFs; 

•  Require  and  incentivize  competition  throughout  program  life  cycles; 

•  Establish  meaningful  metrics  to  assess  OSA  progress; 

•  Develop  the  OSA  Strategy  Execution  Guidebook; 

•  Educate  the  Naval  Engineering,  Logistics,  and  Acquisition  Workforce;  and 

•  Codify  knowledge,  skills  and  processes  in  the  “OSA  Program  Managers  Guidebook,  rev  1.0” 

Phase  2  Tasks 

Tasks  for  phase  2  are  TBD;  here  are  a  few  categories: 

•  Communications  processes  to  provide  transparency  across  PEOs  and  SYSCOMS  about 
existing  programs 

•  Incentives  for  collaboration  and  cooperation 

•  Funding  techniques  for  cross- Enterprise  co-development 

•  Update  the  “OSA  Program  Managers  Guidebook” 

•  Build  on  education  efforts  through  DAU  and  integrate  the  new  Guidebook  with 
standing  courses 


Phase  3  Tasks 

Tasks  for  phase  3  are  TBD;  here  are  categories: 

•  Fine  tune  communications  processes  to  provide  transparency  across  PEOs  and  SYSCOMS 
about  existing  programs,  as  needed 

•  Adjust  incentives  for  collaboration  and  cooperation  as  needed 

•  Add  courses  to  fill  needed  knowledge  gaps 

•  Adjust  funding  techniques  for  cross- Enterprise  co-development 


Assistant  Secretary  of  the  Navy  for  Research ,  Development  and  Acquisition 
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Establish  a  Limited  Number  of  Technical  Reference  Frameworks  (TRFs) 

A  Technical  Reference  Framework  (TRF)  is  an  integrated  set  of  components  that  provide  a 
reusable  architecture  for  a  family  of  related  applications.  TRFs  should  be  capability-based  to 
maximize  employment  and  capability  insertion  on  multiple  platforms.  Limiting  the  number  of 
TRFs  will  increase  interoperability  and  reuse  opportunities,  leading  to  life  cycle  cost  savings. 


Maintaining  non- duplicative  TRFs  will  require  cooperative  interaction  and  create 
interdependencies  across  program  boundaries.  TRFs  are  dynamic  so  will  continue  to  evolve  as 
technology  dictates.  Configuration  management  and  attribute/ characteristic  alignment  processes 
will  be  crucial  TRF  enablers.  To  develop  and  maintain  TRFs,  the  Naval  Enterprise  shall  take  the 
following  steps: 


1.  Analyze  existing  system  TRFs  and  develop  a  detailed  set  of  proven  Enterprise  attributes, 
including  standardized  specifications,  architectures,  data  models,  interoperability  protocols, 
and  software  development  tools; 

2.  Catalog  features  and  suitability  for  a  variety  of  platform  types; 

3.  Promote  tailor-able  open  standards  relative  to  TRF  attributes; 

4.  Coordinate  cross-program  TRF  implementations  to  reduce  duplication  through  transparency; 

5.  Identify,  publish,  and  manage  TRF  elements  necessary  to  move  programs  to  coordinated 
product  lines  and  S&T  investments  using  enterprise-level  TRF  attributes;  and 

6.  Require  PEOs  and  Systems  Commands  to  use  TRFs  for  all  development  unless  explicitly 
waived  by  ASN  (RD&A). 

Change  Acquisition  Processes  to  Adopt  OSA  TRFs 

Changes  must  be  made  to  current  Naval  acquisition  processes  to  allow  Enterprise  adoption  of 
OSA  TRFs.  The  Naval  Enterprise  shall  take  the  following  steps: 

1.  Charter  cross  PEO  groups  and  Communities  of  Interest  (COIs)  through  Program 
Management  Offices  (PMOs)  to  steer  the  development  of  common  TRFs,  applications,  and 
testing  strategies; 

2.  Identify  best  practices  and  collaborative  forums  to  increase  the  likelihood  of  transitioning 
maturing  technology  into  programs  of  record; 

3.  Change  guidance,  procedures,  and  instructions  to  require  preference  for  OSA  implementation 
and  systematic  reuse  for  cost  savings  across  system  life  cycles;  and 

4.  Insert  OSA  into  the  System  Engineering  Technical  Review  (SETR)  and  acquisition  program 
Gate  Review  processes. 

Require  and  Incentivize  Competition  throughout  Program  Life  Cycles 

The  Navy  values  innovation  and  lower  costs  at  all  acquisition  phases  (i.e.,  concept  development, 
design,  build,  maintenance  and  upgrade)  and  system  levels  (i.e.,  component,  system,  platform,  and 
system  of  systems).  The  Naval  Enterprise  shall  take  the  following  competition-focused  steps: 


1.  Create  contract  language  templates  for  use  in  contract  solicitations  at  the  platform, 
integrator,  system,  and  component  levels; 

2.  Develop  tools  and  methods  to  promote  competition  at  the  component  level  and  to 
objectively  measure  the  openness  of  development  environments; 

3.  Require  Program  Managers  to  evaluate  movement  away  from  monolithic  acquisitions  to 
multiple,  modular  acquisitions  enabled  by  OSA; 

4.  Require  Program  Managers  to  secure  and  exercise  data  rights  needed  to  ensure 
future  competition  for  sustainment  maintenance,  and  capability  insertion;  and 

5.  Establish  reward  mechanisms  for  programs  and  personnel  successful  in  achieving 
OSA  implementations  that  rapidly  integrate  innovation  and  lower  total 
ownership  costs. 
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Establish  Meaningful  Metrics  to  Assess  Progress 

Development  and  adoption  of  metrics  that  are  objective,  readily  obtained  (ideally  from  existing 
sources),  easy  to  interpret,  and  actionable  to  enforce  desired  behaviors  (i.e.,  increased  competition, 
component  reuse  and  reduced  costs)  are  vital  to  the  OSA  strategy.  The  Naval  Enterprise  shall  take 
the  following  steps  to  implement  an  OSA  metrics  program: 

1.  Establish  a  set  of  metrics  for  use  in  assessing  the  Enterprise  value  of  new  capabilities; 

2.  Pilot  these  metrics  to  selected  COIs/Programs  of  Record  (PORs)  from  each  domain; 

3.  Update  metrics  based  on  these  pilots  for  application  across  the  Naval  Enterprise; 

4.  Implement  an  Enterprise  metrics  program  and  conduct  periodic  peer  review  assessments 
on  a  sampling  of  PORs  from  across  the  Enterprise;  and 

5.  Identify  patterns  of  strengths  and  weaknesses  in  Enterprise  OSA  implementation  and  apply 
remediation  throughout  program  life  cycles. 

Develop  the  OSA  Strategy  Execution  Guidebook 

The  Execution  Guidebook  will  contain  actionable  steps  for  each  implementation  phase  of 
the  OSA  strategy.  It  will  contain  recommended  changes  in  the  business  model  and  technical 
framework  elements  that  will  begin  by  improving  competition  and  ultimately  result  in  fewer 
programs  that  cost  less  and  deliver  capability  more  rapidly. 

Educate  the  Naval  Engineering,  Logistics,  and  Acquisition  Workforce 

The  success  of  the  OSA  Strategy  depends  heavily  on  a  competent  innovative,  and  well  educated 
workforce.  The  Naval  Enterprise  must  produce  aworkforce  that  is  well-versed  in:  identifying  and 
managing  cross-domain  and  life  cycle  dependencies,  understanding  and  responding  to  adverse 
vendor  behaviors,  ensuring  that  competition  yields  the  desired  results,  and  incorporating  OSA  best 
practices  as  an  integral  part  of  program  management  The  Naval  Enterprise  shall  take  the  following 
steps  to  develop  an  OSA  workforce: 

1.  Target  timely  OSA  training  and  communication  to  optimize  acquisition  program  adoption; 

2.  Develop  training  and  communication  materials,  leveraging  existing  training  materials,  use 
cases,  and  delivery  mechanisms  when  possible; 

3.  Establish  OSA  transparency  mechanisms  to  enable  the  acquisition  workforce  to  become 
aware  of  opportunities  for  collaboration; 

4.  Work  with  the  Defense  Acquisition  University  (DAU)  to  develop  an  Acquisition  OSA 
Qualification  Standard; 

5.  Develop  training  materials  and  methodologies  to  train  the  non  Defense  Acquisition 
Workforce  Improvement  Act  (DAWIA)  Naval  workforce  involved  in  engineering,  logistics, 
and  program  management;  and 

6.  Establish  an  OSA  mentoring  program  for  acquisition  professionals. 
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